MANAGEMENT OF SATELLITE REMOTE SENSING DATA
Robert Denize
Chief Engineer
Australian Centre for Remote Sensing
1992
ABSTRACT
Australian Centre for Remote Sensing (ACRES), receives, catalogues, archives, processes, and distributes Remote Sensing data from Earth Resource Satellites of Landsat 2, 3, 4 & 5, SPOT 1 & 2, ERS-1, JERS-1, and others soon to be added. Ten thousand reels of magnetic tape hold this data in an archive which is growing at two thousand reels a year. It was decided several years ago to take advantage of an optical way of recording this data that gave very high storage density and very long term archival storage, and so be ready for the data deluge of the 1990s.
HISTORY AND BACKGROUND
In September of 1979, a gathering of data scanned by the Earth Resource Satellites Landsat 2 and Landsat 3, began at Alice Springs.
This was the start of the Australian direct collection of this type of data as previously the data collected over Australia had to be purchased from an overseas satellite agency, always assuming that this data had been collected in the first place.
This gathered data. was from an instrument carried aboard the Landsat Satellites called the MultiSpectral Scanner (MSS).
To gather this data, an Agreement had to be in place with the Satellite owners, and a Ground Station constructed to receive and record the data carried on a direct radio link from the Satellite.
The MultiSpectral Scanner data, already converted into digital form on board the satellite, was spun onto magnetic tape, inch wide and 4 600 feet long. This tape now became the vehicle to take the MSS data, containing a strip of Australia land surface information, by air freight to a processing facility set up in Canberra.
THE CATALOGUE
Now the Data Management and Handling starts, for to make this data available to a potential customer, not only has the data to be transformed into a shape useable to that customer, but what data is actually available must be made visible. That is, a catalogue of the available data must be created. Not just a tabular catalogue listing the packets or frames of data available with an estimate of data quality and cloud cover, but also a companion catalogue, which in Remote Sensing using image scanners, takes the form of micro-images of the earth’s surface. These micro-images are presented on micro-fiche, and are arranged to give pictures of the data in standard frames over the total land area and ocean economic zone of Australia.
Thus, for every scene gathered in the ACRES archive, a tabulated catalogue of the holding is available on PC floppy disks, and a companion catalogue of the same holding but showing a subsampled image of each frame is available on micro-fiche.
THE ARCHIVE
Having processed the reel of magnetic tape to extract a Catalogue, the reel now is stored in an environmental controlled vault awaiting the call of a customer to order the data it contains.
These vaults, naturally called the Archive at ACRES, are treated as high security because the data contained therein on reels on Magnetic Tape are unique and are to be kept indefinitely as a record of the surface area of Australia, dating back to 1979.
No back-up of data contained on these reels of Magnetic Tape is made.
THE DATA DOWNPOUR
The MiltiSpectral Scanner on the early Landsat’s have proved such a successful way of digitising the earths surface that it has been continued on the successive Satellites Landsat 4 and 5. From September 1979 to the present, this scanner has scanned the total areal surface of Australia every two weeks, recorded at Alice Springs, and archived at Canberra, more than 8 000 reels worth.
Today there is an armada of instruments carried on various Earth Remote Sensing Satellites; Thematic Mapper on USA Landsats 4 & 5, High Resolution Visible of French SPOT 1 & 2, Short Wave Infra Red and Very Near Infra Red scanners on Japanese ERS-1, and Synthetic Aperture Radar on European ERS-1. All these instruments are scanning the earths surface in finer resolution, more electromagnetic spectral slices, and of varying repeat time intervals.
Alice Springs tracks these Satellites, and daily records five reels of magnetic tape of the type referred to as High Density Digital Tape.
THE GROWING ARCHIVE
Thus, the Archive at Canberra is now growing at almost 2 000 reels of High Density Digital Tape per year, and the situation is set to get worse as new satellites are launched this year to replace and supplement those already in orbit. These replacement satellites contain scanners with more data to be gathered thus more High Density Digital Tapes are used for recording and archiving, more cataloguing to keep trail of, an untenable situation.
"New Remote sensing instrumentation is being developed that will have a volume amount that will dwarf what is presently acquired".
THE PROBLEM
The volume of the Archive at ACRES amounts to 10 000 High Density Digital Tapes, growing at 2 000 per year. The cost of acquiring and maintaining these Tapes, the archive environment, the real estate, and the uniqueness of the data all conspire to force a solution.
"Typically, a major technical innovation is not pushed from behind by a scientific idea, but is pulled forward by a human need".
Satellite Remote Sensing data archiving and retrieval amounts to a need to compact the volume of storage and to refresh the archival medium in a very urgent way. ACRES was faced with losing much of the data gathered as, in spite of the environmental control of its archive, it was discovered that the Magnetic tapes older than about seven years were degrading to the point where it was not possible to read them without special treatment.
Thus, in 1988, a series of studies were started to look at alternatives to the use of High Density Digital Tapes for the long term storage and retrieval of all the Remote Sensing data gathered and held in ACRES archives.
Several considerations were mandated;
- Cost of the medium;
- Cost of the overhead to transfer all the existing archive onto a refresh medium (with the realisation that this would have to be done periodically);
- Cost of the archive vault (real estate, environmental control, fire protection especially the cost of replacing the halon system);
- Cost of maintaining the archived data integrity.
THE ALTERNATIVES
The machines used at Alice Springs to record the Satellite data are of the type called Instrumentation Recorders. These use inch wide magnetic tape on open reels, with the data recorded on longitudinal tracks along the length of the tape. These High Density Digital Recorders have steadily improved to enable higher density of data packing as well as high data rates to match the rates transmitted from the Satellites tracked (up to 105 Mega bits per second).
Other recording techniques have come onto the market, especially recorders based on the techniques evolved in the television recording field using helical scan methods to increase the data packing density even further. This, combined with improvement in magnetic tape formulations, meant that several machines using magnetic tape met the criteria of replacement machines to move the ACRES archive onto.
Indeed, several other ground stations have come out on the side of retaining their archive on magnetic tape by using helical scan recorders combined with juke box stackers to give easy access to large volumes of data in a cost effective way.
Another medium on the market is Optical Disk. These have held promise for many years as a possible medium for storage of large volumes of data. However, the sheer volume of Remote Sensing Archive data is daunting, and when the sums are done, many thousands of Optical disks are required to hold the ACRES Archive, and even with large juke boxes the task is just too large to consider Optical discs. (But we shall see Optical discs in the shape of CD-ROM emerge as a replacement to the Image Catalogue on micro-fiche).
OPTICAL TAPE RECORDING
Out of the need to store very large volumes of digital data for lcng periods of time, and have easy access to this data, a Company in Canada developed a means of using Optical Tape for this purpose.
The idea of using Optical Tape to record digital data was not new and had been tried before with limited success. However, it was a technology whose time had come.
To enable Optical recording and playback, two essential components of development had to come together,
1. the develcpment of lasers, especially the high power solid state laser,
2. the development of sophisticated error correcting methods to enable data to be fully recovered error free over areas of the optical medium having spot sizes 1 micron in diameter obscured by dust, scratches, etc.
The Optical Tape Recorder (OTR) met most of the conditions mandated above, and in the last quarter of 1991 ACRES took delivery of its first Optical Tape Recorder.
In order to understand the significance and potential in this step change in recording and archival storage and retrieval, a discussion of the actual OTR and Optical Tape recording medium is necessary.
THE CREO OPTICAL TAPE RECORDER
CREO, a company based in Burnaby, B.C. Canada, set out in 1984 to develop an Optical Tape Recorder. The essential components of CREO's OTR that produced a practical machine are:-
The Scanner
CREO has developed an air bearing linear scanner which transverses the Optical tape. The scanner writes or reads data by scanning laser beams across the tape when the tape is held stationary.
The Laser
The solid state laser with power output in the 10 milliwatt range focussed to produce a 1.0 micron spot size.
CREO uses an array of diode lasers to lay down 32 data tracks simultaneously, with spot spacing of 1.5 microns.
The Transport System
CREO has developed a precision tape transport system. The tape path never allows the active side of the tape to touch any of the mechanical structure.
The transport system is an open loop predictive-servo capable of moving the tape in small steps to an accuracy of 2 microns, and moving an 860 metre length of tape from one end to the other in less than 60 seconds.
Data Format
Physical records are laid across the tape in swaths containing 32 data tracks and a control track. As the linear scanner crosses the tape, it writes an 80 kilobyte record made up of 16 kilobytes of error correction and 64 kilobytes of user data.
Error Correction
The CREO recorder uses a Reed-Solomon error correction code, heavily interleaved in order to avoid burst or cluster errors. All data is encoded during the write operation and corrected during the read. It is very difficult to lose data, scratches, dust particles, even small holes in the tape will not result in any data loss. The system reconstructs the missing data with an accuracy of 1 in 10 to the power of 12.
An essential measure provided from an archival integrity point of view, is that every time any portion of a tape is read, the recorder reports the amount of data reconstruction needed, and thus provides an excellent record of the error growth allowing the archive manager to decide to recopy the data in question long before data reconstruction fails.
THE ICI 1012 OPTICAL TAPE
An essential component to an Optical Tape recording is the Optical Tape itself. To date, the only company manufacturing and delivering optical tape is ICI of UK.
A second source is expected to come on to the market from Southwall Technologies of Palo Alto, California. Southwall and Dow joined to develop Dow's patented alloy system into manufacturable' optical tape in late summer of 1991. To date only test samples have been delivered by Southwall.
The Optical Tape used on the CREO OTR consists of up to 860 metre length of 35 mm wide tape wound onto a precision open reel. This length of tape will hold one Terrabyte of data.
Optical tape is made up of layers built on a polyester substrate. Of the number of coatings applied, a single recording layer of dye polymer is responsible for the actual reflectance change representing a bit of data.
COST/PERFORMANCE FEATURES
The Ccst/Performance Features of ICI 1012 Optical Tape on the CREO 1003 OTR are almost too good to be true.
Low media cost; less than 10 cent / megabyte and falling with time.
High Data Rate; 0 up to 3 Megabyte / second.
Volumetric efficiency; factor 10 higher than advanced helical magnetic tape.
Unlimited read; greater than 40 000 rewind cycles.
Long media life; greater than 30 years in the presence of corrosive gases typical of the standard office environment.
Tape rewind periods; well in excess of magnetic media requirements, thus allowing reduced archive maintenance costs.
ACRES PUT TO USE OPTICAL TAPE RECORDERS
ACRES have taken delivery, in 1991 and 1992, three Optical Tape Recorders and has started the massive task of transcribing the existing Archive onto Optical Tape.
The Transcription Process
The transcription process is complex in that it must preserve the integrity of the data as it is copied onto the new media.
As part of the transcription process error detection has been built into the data transfer path that monitors the error detected in the data and brings it to the notice of the transcription operator.
Should the error be correctable (the usual source of error is contamination on the magnetic tape recorder read head), the operator may choose to re-run the transcription. Should the error be absolute, then this can be noted in the catalogue data base, or it can be chosen not to transcribe that magnetic tape and thus delete the entry from the refreshed data base.
The Catalogue update
The published catalogue of ACRES archive has been built up as the data has been acquired. Over the years, a percentage of the archive has been lost (due to accident, magnetic tape deterioration, removal of unsatisfactory or unreadable data, etc). As the archive is refreshed onto Optical Tape, the ACRES catalogue has the opportunity to be refreshed as well so that it now reflects what is now on Optical Tape, thus what is truly stored in the ACRES archive.
Thus as part of the transcription process, the ACRES data catalogue is extracted and replaced with the data held on Optical Tape.
This data base has to be updated as well to inform the processor on which Optical Tape and where that the data is held for the processing of a customer order.
Transcription Order
Although it is possible to arrange the transcription in any order on the Optical Tape, ACRES decided that it would be sensible to retain the chronological order of the data gathered, and to reserve separate Optical Tape reels for the separate instruments recorded.
The Volume Reduction
The transcription is labour intensive when it is considered how many High Density Digital Tapes will go to fill one Optical Tape.
MultiSpectral Scanner data
One Optical tape holds about 850 satellite passes of MSS data. This translates into 850 High Density Digital Tapes, each 4 600 feet long, that can be stored onto one Optical Tape. Expressed another way, one 4 600 foot reel of magnetic tape holding MSS data fill about one metre length of Optical Tape, a dramatic reduction of physical volume in anybodies language.
Thematic Mapper data
This instrument supplements the MSS with finer pixel resolution and more spectral bands. The data volume per satellite pass is much greater than MSS, and the High Density recorders used at Alice Springs put data onto the High Density Digital Tape in a higher density and on greater length reel of 7 200 feet.
Nevertheless, about 150 of these tapes can be stored onto one Optical Tape, or one magnetic tape holding a satellite pass of Thematic Mapper data fill a little over five metres of Optical Tape.
RECYCLING
To date, ACRES have filled three Optical Tapes with MSS data, and four with Thematic Mapper data. This has freed up about 2,500 High Density Digital Tapes that have been used to recycle to Alice Springs thereby making a significant saving in magnetic tape purchases, and freeing up valuable shelf space in the archive vault.
THE FUTURE
The first priority at ACRES is to transfer the magnetic archive onto Optical Tape. This is now occurring with the MSS and the Thematic Mapper data.
The next part of the archive to be transcribed will be Synthetic Aperture Radar data of ERS-1, followed by other data forms as progress is made in the archive transfer.
It is expected to take about two years to transfer all of the archive onto Optical Tape, with another year to transfer that data gathered in the meantime unless arrangements are made to transfer data as it is acquired at Alice Springs.
The total archive is expected to fit onto about 30 reels of Optical tape.
The next major phase will be the duplication of the Optical Tape archive.